Enhanced fluorescence imaging for imaging system

ABSTRACT

A fluorescence imaging system is configured to generate a video image onto a display. The system includes a light source for emitting infrared light and white light, an infrared image sensor for capturing infrared image data, and a white light image sensor for capturing white light image data. Data processing hardware performs operations that include filtering the infrared image data with a first digital finite impulse response (FIR) filter configured to produce a magnitude response of zero at a horizontal Nyquist frequency and a vertical Nyquist frequency. The operations also include filtering the infrared image data with a second digital FIR filter configured with a phase response to spatially align the white light image data with the infrared image data. The operations also include combining the white light image data and the infrared image data into combined image data and transmitting the combined image data to the display.

TECHNICAL FIELD

The disclosure relates to an enhanced fluorescence imaging system formedical procedures.

BACKGROUND

Endoscopes are commonly used to provide access to body cavities whiledecreasing the invasiveness of a surgical procedure. The endoscope mayinclude one or more light sources that emit both white (i.e., visible)light and infrared light. The white light is typically used as areference light or illuminating light, while the infrared light istypically used as an excitation light. That is, the infrared light isused to irradiate medication (e.g., dye) administered to a patient withinfrared light, which in turn causes the medication to emit fluorescencelight. The endoscope includes one or more image sensors to capture thereflected white light and/or the emitted fluorescence light. However,often the quantum efficiency of these image sensors is lacking, thuscausing the sensitivity and contrast of the images to suffer. Whileimage sensors with higher quantum efficiencies are available, their usein endoscopy is limited due to issues such as sensitivity, phasemisalignment, and contrast loss.

SUMMARY

One aspect of the disclosure provides a system for enhanced fluorescenceimaging. The system includes a light source for emitting infrared lightand white light, an infrared image sensor for capturing infrared imagedata in the infrared spectrum, and a white light image sensor forcapturing white light image data in the white light spectrum. The systemalso includes data processing hardware in communication with theinfrared image sensor and the white light image sensor and memoryhardware in communication with the data processing hardware. The memoryhardware stores instructions that when executed on the data processinghardware cause the data processing hardware to perform operations thatinclude filtering the infrared image data with fluorescence imageenhancer having a first digital finite impulse response (FIR) filter anda second digital FIR filter. The first digital FIR filter is configuredto produce a magnitude response of zero at a horizontal Nyquistfrequency and a vertical Nyquist frequency. The second digital FIRfilter is configured with a phase response to spatially align the whitelight image data with the infrared image data. The operations alsoinclude combining the aligned white light image data and the aligned andfiltered infrared image data into combined image data and transmittingthe combined image data to the display.

Implementations of the disclosure may include one or more of thefollowing optional features. In some implementations, the fluorescenceimage enhancer further includes a third digital FIR filter that isconfigured to equalize for contrast loss from fluorescence diffusion. Insome examples, the fluorescence image enhancer further includes a fourthdigital FIR filter that is configured to adjust a sensitivity of theinfrared image data and a contrast of the infrared image data based onreceived infrared light. The fourth digital FIR filter may include a lowpass filter with a cutoff frequency and a magnitude. Optionally, thefourth digital FIR filter is further configured to increase thesensitivity of the infrared image data and decrease the contrast of theinfrared image data when the received infrared light is below athreshold level.

In some implementations, the fourth digital FIR filter adjusts theinfrared image data by increasing the sensitivity of the infrared imagedata and decreasing the contrast of the infrared image data when thereceived infrared light is below a threshold level. The fourth digitalFIR filter may be further configured to increase the sensitivity of theinfrared image data by increasing the magnitude and decrease thecontrast of the infrared image data by decreasing the cutoff frequencywhen the received infrared light is below a threshold level. The fourthdigital FIR filter, in some examples, adjusts the infrared data bydecreasing the sensitivity of the infrared image data and increasing thecontrast of the infrared image data when the ambient light level isabove a threshold level. The fourth digital FIR filter may be furtherconfigured to decrease the sensitivity of the infrared image data bydecreasing the magnitude and increase the contrast of the infrared imagedata by increasing the cutoff frequency when the received infrared lightis below a threshold level.

In some implementations, the fourth digital FIR filter is configured toadjust the sensitivity and the contrast of the infrared image data byadjusting an accumulation of light in a neighborhood of the infraredimage data. The infrared image sensor and the white light image sensormay have the same resolution. The infrared image sensor, in someexamples, includes a near infrared image sensor. Optionally, theinfrared image sensor and the white light image sensor comprise a Bayercolor filter array.

In some implementations, the system further includes a dichroic prism.The dichroic prism splits received light into white light and infraredlight and the infrared image sensor receives the infrared light from thedichroic prism and the white light image sensor receives the white lightfrom the dichroic prism. The first digital FIR filter may be furtherconfigured to produce a flat magnitude response prior to the horizontalNyquist frequency and the vertical Nyquist frequency. The second digitalFIR filter may be configured to spatially adjust a position of theinfrared image data in two dimensions.

The details of one or more implementations of the disclosure are setforth in the accompanying drawings and the description below. Otheraspects, features, and advantages will be apparent from the descriptionand drawings, and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments set forth in the drawings are illustrative and exemplaryin nature and not intended to limit the subject matter defined by theclaims. The following description of the illustrative embodiments can beunderstood when read in conjunction with the following drawings, wherelike structure is indicated with like reference numerals and in which:

FIG. 1 is a schematic view of an example system for enhancedfluorescence imaging.

FIG. 2 is a perspective view of a known Bayer color filter array imagesensor.

FIG. 3 is a graph of quantum efficiency versus wavelength for knownimage sensors.

FIG. 4 is a graph of quantum efficiency versus wavelength for the imagesensor of FIG. 2 .

FIG. 5A is a graph of exemplary diffusion loss spatial frequencyresponse for varying depths of tissue.

FIG. 5B is a graph of exemplary matched finite impulse responsecorrection filters for the varying depths of tissue of FIG. 5A.

FIG. 5C is a graph of corrected frequency responses for the varyingdepths of tissue of FIGS. 5A and 5B.

FIG. 6 is a plot of a spatial frequency response of a 2×2 binningprocess.

FIG. 7 is a plot of a spatial frequency response of an adaptive virtualbinning process in accordance with the present disclosure.

FIG. 8 is a flowchart of an example method enhancing fluorescenceimaging.

FIG. 9 is a schematic view of an example computing device that may beused to implement the systems and methods described herein.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Implementations herein are directed toward an enhanced fluorescenceimaging system that reconstructs a fluorescent dye image from colorfilter array (CFA) channels with minimal CFA modulation and maximallyflat spatial frequency response (SFR). The imaging system improvessensitivity of the infrared image sensor output over previous systems,while adaptively controlling the sensitivity improvement by using anextended precision parametrically controlled two-dimensional (2D)low-pass filter. The imaging system may also correct for misregistrationof the fluorescing dye and white light prism output images and correctfor diffusion-based contrast loss in the image. As used herein, the term“fluorescent dye” refers to dye approved for medical use that isconfigured to reflect infrared light such as Indocyanine Green (ICG).

Endoscopy, a nonsurgical medical procedure used to examine internal bodycavities (e.g., a digestive tract), is increasingly used as an effectivediagnostic tool. The procedures are typically performed usingendoscopes, which include, in their most basic form, a flexible tubewith a light source and a camera. The flexible tube is passed through anorifice (e.g., the mouth) of a patient and the camera records imagesilluminated by the light.

In addition to visible white light, many endoscopes are capable ofemitting other spectrums of light. For example, it is common forendoscopes to also emit infrared light to support fluorescent imaging.ICG is a cyanine dye used frequently in medical diagnostics andendoscopy for determining cardiac output, hepatic function, liver andgastric blood flow, and for ophthalmic angiography. For example, ICGdistribution within the tissue enables intraoperative evaluation of atissue perfusion and vacuolization, identification of criticalneurovascular structures and differentiation of tissue plains betweenlesions and adjacent structures. ICG has a peak spectral absorption inthe near infrared spectrum at approximately 800 nm. The dye, after beingadministered intravenously, acts as an indicator substance by bindingtightly to plasma proteins within the vascular system. ICG, whenirradiated with light between 750 nm and 950 nm, emits fluorescence. Theendoscope, after irradiating the ICG with the near infrared light,detects and images this fluorescence.

Endoscopes may be equipped with one or more image sensors to image bothwhite (i.e., visible) light and infrared light. For example, someendoscopes are equipped with a three charge-coupled device (3CCD)camera. A 3CCD camera uses a prism to split received light into threebeams, one of which is directed to a red CCD, one to a green CCD, andone to a blue CCD. These imagers tend to have a very low quantumefficiency (see FIG. 3 ). That is, the ratio of incident photonsconverted to electrons may be very low, yielding poor sensitivity.Endoscopes, in some examples, are equipped with multiple image sensorswith each sensor dedicated to a respective frequency band using a filtercommonly known as a Bayer filter (see FIG. 2 ). In some instances, thisallows the image sensors to have a significantly higher quantumefficiency (see FIG. 4 ).

For illustrative purposes, a description of a fluorescence imageenhancer is provided within the context of an endoscopic system 100.However, it should be appreciated that the fluorescence image enhancermay be utilized in other applications, illustratively including anexoscope, borescopes and other systems having two or moreillumination-types and one or more image sensors. Furthermore, althoughthe system is described with respect to medical applications usingfluorescing dye, it should be understood that industrial applicationsusing other combinations of white light and colored light of narrowwavelength ranges may benefit from the same principles.

Referring to FIG. 1 , in some implementations, an endoscopic examplesystem 100 includes one or more light sources 110. The light source 110emits both white light (WL) 112 a and near infrared light (NIR) 114 a(sometimes referred to as color light). While the light source(s) 110may emit the WL 112 a and the NIR light 114 a simultaneously, typicallythe light source 110 alternates between emitting WL 112 a and NIR light114 a. That is, in some examples, the light source 110 rapidly switchesbetween emitting WL 112 a and NIR light 114 a. The WL 112 a illuminatesthe surgical site of the system 100. The light source 110 may includeone or more light-emitting diodes (LEDs) or any other appropriatelight-emitting device. Separate light sources may emit the WL 112 a andthe NIR light 114 a respectively. In some examples, the light source 110is included within a camera head unit 102.

Light 112 a, 114 a emitted by the light source 110 travels along a lightguide 116 (e.g., an optical fiber) and, after exiting the light guide116, illuminates or irradiates a target area 10 (e.g., an internalcavity of a patient). Reflected WL 112 b (i.e., WL 112 a that hasreflected from the target area 10) and emitted fluorescent light (FL)114 b (i.e., light emitted by, for example, ICG that has been irradiatedby NIR light 114 a) is directed back through the light guide 116 to adichroic prism 120. The dichroic prism 120 splits received light intotwo beams of differing wavelength. That is, the dichroic prism 120splits the received light, which may include reflected WL 112 b and/orFL 114 b, to image sensors 130 a, 130 b. For example, any reflected WL112 b (i.e., visible light) that passes through the prism 120 may bedirected to the WL image sensor 130 a, while any FL 114 b that passesthrough the prism 120 may be directed to the FL image sensor 130 b(i.e., light with a wavelength between 800 nm and 1200 nm). In someexamples, the prism 120 and image sensors 130 are also included withinthe camera head unit 102.

The image sensors 130 may be a complementary metal oxide semiconductor(CMOS) or a Charged Coupled Device (CCD). It should be appreciated thatany pixelated image sensor 130 currently known or later developed may bemodified and adopted for use herein. The image sensors 130, in someimplementations, include color filter arrays (CFAs). Referring now toFIG. 2 , the image sensors 130 may include Bayer CFAs (sometimesreferred to as a Bayer filter). Bayer CFAs include a mosaic CFA forarranging red, green, and blue color filters on a grid of photosensors.As illustrated in FIG. 2 , the filter pattern is 50% green, 25% red, and25% blue, as human eyes are most sensitive to green light. Thus, eachpixel is filtered to record only one of the three colors, and variouswell-known de-mosaicking algorithms are used to obtain full-colorimages. In some examples, the WL image sensor 130 a and the FL imagesensor 130 b are different sensors with the same or differentresolutions. In other examples, the sensors 130 are identical sensors.Identical sensors (e.g., the same resolution, geometry, etc.) oftenimproves and eases manufacturing, assembly, and alignment of the system100.

The dual image sensors 130 may exhibit greatly increased quantumefficiency relative to a, for example, three (3) CCD RGB prism assembly.FIG. 3 illustrates the quantum efficiency of the three (3) CCD RGB prismassembly, which exhibits a quantum efficiency of only 8% with bluechannel out-of-band transmission at the ICG fluorescence centerwavelength of 830 nm. That is, the illustrated assembly only capturesapproximately 8% of the photons that the sensor receives from the ICGfluorescence. In contrast, FIG. 4 illustrates the quantum efficiency ofnearly 24% of the prism 120 and image sensors 130 of the presentdisclosure. In addition, FIG. 4 illustrates a near neutral response ofthe sensors' output at the ICG fluorescence center wavelength of 830 nm.

With reference again to FIG. 1 , the sensors 130 transmit WL data 132 aand FL data 132 b to a camera control unit (CCU) 140. The CCU 140 may,in some examples, be included within the camera head unit 102, while inother examples is remote from the camera head unit 102. The CCU 140includes computing resources 142 (e.g., data processing hardware) andstorage resources 144 (e.g., memory hardware). In some implementations,the CCU 140 is disposed physically at the system 100 (e.g., within thecamera head unit 102) and in wired communication with the sensors 130.In other implementations, the CCU 140 is in wireless communication withthe sensors 130 (e.g., via wireless, Bluetooth, etc.) and may be remotefrom the sensors 130 and/or system 100. In this case, the CCU 140 maycorrespond to any appropriate computing device 900 (see FIG. 9 ), suchas a desktop workstation, laptop workstation, or mobile device (e.g.,smart phone or tablet). In yet other implementations, the data 132 maybe stored in nonvolatile storage at the system 100 (e.g., a thumb drive)and later removed to be processed at data processing and memory hardware140, 142 remote from the sensors 130.

The data processing hardware 142 executes (i.e., using instructionsstored on the storage resources 144) a fluorescence image enhancer 150.In some implementations, the fluorescence image enhancer 150 executesone of a plurality of finite impulse response (FIR) filter. A FIR filterhas an impulse response of finite duration (i.e., the output willeventually settle to zero). FIR filters require no feedback and areinherently stable and easily provide a linear phase response, makingthem ideal for phase-sensitive applications such as image processing.One of the FIR filters may be a diffusion contrast-loss equalizationfilter 200. In some examples, the diffusion contrast-loss equalizationfilter 200 is a digital FIR filter. The diffusion contrast-lossequalization filter 200 compensates for contrast loss due to, forexample, ICG diffusion. As ICG naturally diffuses, the contrast (i.e.,the difference in color that allows objects to be distinguishable)decreases, which in turn decreases the effectiveness of the diagnosticnature of the image.

In addition to this natural diffusion, the depth of view into the tissuealso diffuses the dye. That is, the deeper into the tissue that isimaged, the more diffused the ICG. For example, for every millimeter oftissue depth, image contrast may be reduced. This diffusion (i.e., lossof contrast) may be represented as a Gaussian blur. For example, thesurface of the tissue may be represented as a point spread function, andas the viewpoint moves deeper into the tissue, the Gaussian blurincreases. Referring now to FIG. 5A, diffusion loss spatial frequencyresponse is illustrated for four exemplary and arbitrary tissue depths.As the depth increases, the maximum spatial frequency decreases. ThisGaussian blur (i.e., the ICG diffusion) has a Gaussian impulse responseand Gaussian frequency response that may be matched by the diffusioncontrast-loss equalization filter 200. That is, the filter 200 mayinvert the Gaussian blur (e.g., the frequency response) to increase thecontrast of the ICG. Referring now to FIG. 5B, the frequency responsefor four matched FIR correction filters for the arbitrary depthsprovided in FIG. 5A are illustrated. In some examples, the correctionfilter(s) 200 are gain limited to approximately 10 to 30 dB (e.g., 20dB). FIG. 5C illustrates the cascade of the contrast loss spatialfrequency response and the FIR correction filter for each exemplarydepth. In some implementations, the system 100 includes a single filter200 with adaptive parameters. In other implementations, the systemincludes a plurality of filters 200 with different parameters (e.g.,frequency responses) that may be automatically selected by the system100 (e.g., based on input measurements such as ambient light levels orbased on image analysis) or manually selected by a user. For example,the user may access a menu and select from among the different filters200 depending on the amount of natural diffusion (i.e., based on atemporal nature) or the depth of the imaging within the tissue. Eachfilter 200 may select for a desired intensity of contrast.

As stated above, the fluorescence image enhancer 150 executes one ormore FIR filters. For example, the fluorescence image enhancer 150 mayalso execute a WL-FL co-site alignment digital FIR filter 300. The imagesensors 130 are typically fixed in place and the prism 120 directs lighttoward each sensor 130. The placement of the sensors 130 must be veryprecise to keep the white light and IR light spatially aligned (i.e.,the sensors must be placed exactly where the prism directs the light).Such precise alignment is difficult to obtain and, after separationthrough the prism 120 (e.g., because of minor misalignment ormisplacement of the image sensors 130), the WL data 132 a and the FLdata 132 b may be out of alignment. This misalignment or misregistrationleads to artifacts and defects after combining the data 132 to createthe combined image data 170. To correct this misalignment, the phasealignment filter 300 includes a customized linear phase response thatspatially aligns the WL data 132 a and the FL data 132 b. That is, insome examples, the alignment filter 300 shifts the FL data 132 b (or,alternatively, the WL data 132 a) in space (in one or two dimensions) bylinearly shifting the phase response. Failure to compensate for themisalignment between the image sensors 130 will decrease the sharpness(i.e., increase blurriness) in the combined image data 170.

In some examples, the misalignment of the sensors 130 may be measuredduring manufacturing and the alignment filter 300 may be calibratedafter determining the misalignment. Optionally, the system 100 mayinclude a plurality of alignment filters 300 (i.e., a plurality ofprecomputed filters 300 stored in memory 144) with different parameters(e.g., different linear phase responses) and the system 100 or the usermay select which filter best compensates for the misalignment of thesensors 130. For example, each successive alignment filter 300 mayadjust an image by one half of a pixel horizontally or vertically, or incombination.

In some examples, the fluorescence image enhancer 150 executes anadaptive low light sensitivity digital FIR filter 400 to adaptivelycontrol the sensitivity improvement of the enhancer 150 based onreceived light levels. That is, the adaptive light sensitivity filter400 allows the enhancer 150 to tradeoff between sensitivity and contrastby adjusting the sensitivity and the contrast of the FL data 132 bdepending on lighting conditions. For example, when the received lightlevel is below a threshold level, the adaptive light sensitivity filter400 may decrease the contrast and increase the sensitivity. In someexamples, the threshold may be 10 microwatts per square centimeter persteradian. Similarly, when the ambient light level is above thethreshold level (or a second, different threshold level), the adaptivelight sensitivity filter 400 may increase the contrast and decrease thesensitivity. Because of the nature of medical diagnostic dyes such asICG (i.e., the diffuse nature after binding with the test subject), itis often advantageous to sacrifice sharpness and/or contrast in favor ofsensitivity because the contrast is often of limited value. In someexamples, the system 100 (e.g., at sensor 130 b) measures or determinesan amount of fluorescent light received, and the adaptive lightsensitivity filter 400 decreases contrast and increases sensitivity whenthe measured or determined amount of light is below a threshold amountof watts per square centimeter per steradian. Similarly, the adaptivelight sensitivity filter 400 may increase the contrast and decrease thesensitivity when the amount of light is above the same or differentthreshold amount of watts per square centimeter per steradian.

For example, to increase sensitivity at the expense of contrast (i.e.,filter bandwidth), the adaptive light sensitivity filter 400 mayincrease an amount of accumulation (i.e., increase the sum of one ormore pixels values) over a neighborhood (i.e., a select pixel and agroup or sub-set of pixels that surround the select pixel) of the FLdata 132 b. That is, instead of performing any binning in or at theimage sensor 130 b itself, the filter 400 allows the system 100 toadaptively adjust the amount of light accumulation in a neighborhoodbased on the measured light levels. Binning (as discussed in more detailbelow) refers to the process conducted by some CCD image sensors wherelight from multiple pixels is added together, thereby decreasingresolution but increasing sensitivity. The system may include (i.e., inmemory storage 144), a plurality of filters 400 with different amount ofaccumulations that the system 100 or the user may select from based onFL light levels. For example, the system 100 may include a filter 400that accumulates light at an amount similar to 2×2 binning, 4×4 binning,8×8 binning, 16×16 binning, etc. When the amount of FL light issufficiently high, the system 100 may not apply the adaptive lightsensitivity filter 400 at all. The adaptive light sensitivity filter 400may adjust the amount of light accumulated with a low pass filter thathas a cutoff at a threshold frequency and a magnitude of a frequencyresponse. The tradeoff between the cutoff frequency and the magnitudeadjusts the tradeoff between sensitivity and contrast.

Optionally, the fluorescence image enhancer 150 includes a Nyquistsuppression filter 500. Typically, endoscopes, when processing capturedimage data, will conduct a 2×2 binning process. As briefly describedabove, binning is the common process of combining the values fromadjacent pixels (horizontal and/or vertical) to increase readout speedand improve signal to noise ratio and sensitivity at the expense ofresolution. When conducting a 2×2 binning process, the values capturedby four adjacent pixels are combined together. Typical endoscopes willconduct 2×2 binning, then downsample from ultra high definition (UHD) tohigh definition (HD). The endoscope will then perform any desiredprocessing on the HD image data prior to upsampling back to UHD. Thisresults in contrast-loss SFR errors (from the binning) and aliasing(from the downsampling and upsampling). Referring now to FIG. 6 , thiscommon approach leads to suboptimal spatial frequency response, as thereis significant magnitude loss as the frequency approaches the Nyquistfrequency (i.e., the bottom-right corner of the plot of FIG. 6 ).

In contrast, the Nyquist suppression filter 500 of the fluorescenceimage enhancer 150 provides improvement over the 2×2 (i.e., 2 horizontalby 2 vertical) binning SFR process used by typical endoscopes by insteadusing larger-extent digital FIR filtering with a maximally flatmagnitude response and magnitude response zeros at horizontal andvertical Nyquist (relative to the sampling spectrum of the FL imagesensor 130 b) to remove color filter array modulation. The color filterarray modulation may be removed because the system 100 treats the FLimage sensor 130 b like a monochrome sensor.

Referring now to FIG. 7 , the SFR of the Nyquist suppression filter 500achieves a magnitude of zero at the horizontal and the vertical Nyquistfrequency (i.e., the bottom-right corner of the plot of FIG. 7 ) whilemaintaining a maximum or near maximum frequency response at theremaining frequencies. The filter 500 provides an extended precisionlow-pass magnitude response used as an “adaptive virtual binningprocess.” That is, the filter 500, instead of a fixed 2×2 binningprocess, provides a much finer “virtual binning process”. That is,instead of actual binning on the image sensors 130, the filter 500allows for smooth and dynamic control over the frequency response of theimage data 132.

The fluorescence image enhancer 150, after processing and enhancing theFL data 132 b, outputs the enhanced FL data 160. The enhanced FL data160 may be combined or superimposed with the WL data 132 a to create thecombined image data 170. In other examples, the light source 110switches between emitting WL light 112 a and NIR light 114 a, and thusthe combined image data switches between consisting of WL data 132 a andenhanced FL data 160 (e.g., each frame alternates between WL data 132 aand FL data 160). Thus, the combined image data 170 includes theenhanced FL data 160 and/or the WL data 132 a (which may or may not befurther processed). The combined image data 170 is transmitted to thedisplay 180, which processes the image 170 to generate a visible image(i.e., a picture or video).

Accordingly, the endoscopic system 100 provided may reconstruct abaseband image (e.g., an ICG baseband image) from CFA channels withminimal CFA modulation and maximally flat spatial frequency response(SFR) while improving the sensitivity of the ICG sensor output. Further,the system 100 may adaptively control sensitivity improvement by usingan extended precision parametrically controlled 2D low-pass filter,correct for misregistration of the ICG and WL prism output images, andcorrect for diffusion-based contrast loss in the ICG image. While fourseparate filters 200, 300, 400, 500 are illustrated and described, itshould be appreciated that the four filters 200, 300, 400, 500 may becombined in any combination to result in one to four filters withoutdeparting from the spirit and scope of the disclosure.

FIG. 8 is a flowchart of example operations 800 for a fluorescenceimaging system 100 that is configured to generate a video image onto adisplay 180. The system 100 includes a light source 110 for emittinginfrared light 114 a and white light 112 a, an infrared (or fluorescent)image sensor 130 b for capturing infrared image data 132 b in theinfrared spectrum, and a visible light image sensor 130 a for capturingvisible light image data 132 a in the visible light spectrum. The system100 also includes data processing hardware 142 in communication theinfrared image sensor 130 b and the white light image sensor 130 a.Memory hardware 144 in communication with data processing hardware 142stores instructions that when executed on the data processing hardware142 cause the data processing hardware 142 to perform operations. Theoperations include, at step 802, filtering the infrared image data 132 bwith a first digital finite impulse response (FIR) filter 500 configuredto produce a magnitude response of zero at a horizontal Nyquistfrequency and a vertical Nyquist frequency.

At step 804, the operations include filtering the infrared image data132 b with a second digital FIR filter 300 configured with a phaseresponse to spatially align the white light image data 132 a with theinfrared image data 132 b. At step 806, the operations include combiningthe aligned white light image data 132 a and the aligned and filteredinfrared image data 132 b into combined image data 170, and at step 808,transmitting the combined image data 170 to the display 180. That is, insome examples, both white light image data 132 a and infrared image data132 b is transmitted to the display 180 (e.g., each frame alternatesbetween WL data and FL data) and the display 180 displays somecombination of the WL data 132 a and the FL data 132 b for the user.

FIG. 9 is schematic view of an example computing device 900 (e.g., dataprocessing hardware 142 and memory hardware 144) that may be used toimplement the systems and methods described in this document. Forexamples, computing device 900 may perform tasks such as controlling thelight source 110 (e.g., enabling and disabling the light source,switching between white light and MR light, etc.), configuring andcommunicating with the image sensors 130 (e.g., receiving the imagedata), and implementing and executing one or more of the digital FIRfilters 200, 300, 400, 500. In some examples, the computing device 900transmits image data to the display 180. That is, using the datareceived from the image sensors 130, the computing device 900 may storeand execute instructions or operations to implement any number of thedigital FIR filters 200, 300, 400, 500. The computing device 900 isintended to represent various forms of digital computers, such aslaptops, desktops, workstations, personal digital assistants, servers,blade servers, mainframes, and other appropriate computers. Thecomponents shown here, their connections and relationships, and theirfunctions, are meant to be exemplary only, and are not meant to limitimplementations of the disclosures described and/or claimed in thisdocument.

The computing device 900 (e.g., data processing hardware 142) includes aprocessor 910, memory 920, a storage device 930, a high-speedinterface/controller 940 connecting to the memory 920 and high-speedexpansion ports 950, and a low speed interface/controller 960 connectingto a low speed bus 970 and a storage device 930. Each of the components910, 920, 930, 940, 950, and 960, are interconnected using variousbusses, and may be mounted on a common motherboard or in other mannersas appropriate. The processor 910 can process instructions for executionwithin the computing device 900, including instructions stored in thememory 920 or on the storage device 930 to display graphical informationfor a graphical user interface (GUI) on an external input/output device,such as display 980 coupled to high speed interface 940. In otherimplementations, multiple processors and/or multiple buses may be used,as appropriate, along with multiple memories and types of memory. Also,multiple computing devices 900 may be connected, with each deviceproviding portions of the necessary operations (e.g., as a server bank,a group of blade servers, or a multi-processor system).

The memory 920 stores information non-transitorily within the computingdevice 900. The memory 920 may be a computer-readable medium, a volatilememory unit(s), or non-volatile memory unit(s). The non-transitorymemory 920 may be physical devices used to store programs (e.g.,sequences of instructions) or data (e.g., program state information) ona temporary or permanent basis for use by the computing device 900.Examples of non-volatile memory include, but are not limited to, flashmemory and read-only memory (ROM)/programmable read-only memory(PROM)/erasable programmable read-only memory (EPROM)/electronicallyerasable programmable read-only memory (EEPROM) (e.g., typically usedfor firmware, such as boot programs). Examples of volatile memoryinclude, but are not limited to, random access memory (RAM), dynamicrandom access memory (DRAM), static random access memory (SRAM), phasechange memory (PCM) as well as disks or tapes.

The storage device 830 is capable of providing mass storage for thecomputing device 800. In some implementations, the storage device 830 isa computer-readable medium. In various different implementations, thestorage device 830 may be a floppy disk device, a hard disk device, anoptical disk device, or a tape device, a flash memory or other similarsolid state memory device, or an array of devices, including devices ina storage area network or other configurations. In additionalimplementations, a computer program product is tangibly embodied in aninformation carrier. The computer program product contains instructionsthat, when executed, perform one or more methods, such as thosedescribed above. The information carrier is a computer- ormachine-readable medium, such as the memory 820, the storage device 830,or memory on processor 810.

The high speed controller 840 manages bandwidth-intensive operations forthe computing device 800, while the low speed controller 860 manageslower bandwidth-intensive operations. Such allocation of duties isexemplary only. In some implementations, the high-speed controller 840is coupled to the memory 820, the display 880 (e.g., through a graphicsprocessor or accelerator), and to the high-speed expansion ports 850,which may accept various expansion cards (not shown). In someimplementations, the low-speed controller 860 is coupled to the storagedevice 830 and a low-speed expansion port 890. The low-speed expansionport 890, which may include various communication ports (e.g., USB,Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or moreinput/output devices, such as a keyboard, a pointing device, a scanner,or a networking device such as a switch or router, e.g., through anetwork adapter.

The computing device 800 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as astandard server 800 a or multiple times in a group of such servers 800a, as a laptop computer 800 b, or as part of a rack server system 800 c.

Various implementations of the systems and techniques described hereincan be realized in digital electronic and/or optical circuitry,integrated circuitry, specially designed ASICs (application specificintegrated circuits), computer hardware, firmware, software, and/orcombinations thereof. These various implementations can includeimplementation in one or more computer programs that are executableand/or interpretable on a programmable system including at least oneprogrammable processor, which may be special or general purpose, coupledto receive data and instructions from, and to transmit data andinstructions to, a storage system, at least one input device, and atleast one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms “machine-readable medium” and“computer-readable medium” refer to any computer program product,non-transitory computer readable medium, apparatus and/or device (e.g.,magnetic discs, optical disks, memory, Programmable Logic Devices(PLDs)) used to provide machine instructions and/or data to aprogrammable processor, including a machine-readable medium thatreceives machine instructions as a machine-readable signal. The term“machine-readable signal” refers to any signal used to provide machineinstructions and/or data to a programmable processor.

The processes and logic flows described in this specification can beperformed by one or more programmable processors, also referred to asdata processing hardware, executing one or more computer programs toperform functions by operating on input data and generating output. Theprocesses and logic flows can also be performed by special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit). Processors suitable for theexecution of a computer program include, by way of example, both generaland special purpose microprocessors, and any one or more processors ofany kind of digital computer. Generally, a processor will receiveinstructions and data from a read only memory or a random access memoryor both. The essential elements of a computer are a processor forperforming instructions and one or more memory devices for storinginstructions and data. Generally, a computer will also include, or beoperatively coupled to receive data from or transfer data to, or both,one or more mass storage devices for storing data, e.g., magnetic,magneto optical disks, or optical disks. However, a computer need nothave such devices. Computer readable media suitable for storing computerprogram instructions and data include all forms of non-volatile memory,media and memory devices, including by way of example semiconductormemory devices, e.g., EPROM, EEPROM, and flash memory devices; magneticdisks, e.g., internal hard disks or removable disks; magneto opticaldisks; and CD ROM and DVD-ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, one or more aspects of thedisclosure can be implemented on a computer having a display device,e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, ortouch screen for displaying information to the user and optionally akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's client device in response to requests received from the webbrowser.

While particular embodiments have been illustrated and described herein,it should be understood that various other changes and modifications maybe made without departing from the spirit and scope of the claimedsubject matter. Moreover, although various aspects of the claimedsubject matter have been described herein, such aspects need not beutilized in combination. It is therefore intended that the appendedclaims cover all such changes and modifications that are within thescope of the claimed subject matter.

1.-20. (canceled)
 21. A fluorescence imaging system for use in a medicalprocedure, the imaging system configured to generate a video image ontoa display, the fluorescence imaging system comprising: a light sourcefor emitting infrared light and white light; an infrared image sensorfor capturing infrared image data in the infrared spectrum; a whitelight image sensor for capturing white light image data in the whitelight spectrum; data processing hardware in communication with theinfrared image sensor and the white light image sensor; and memoryhardware in communication with the data processing hardware, the memoryhardware storing instructions that when executed on the data processinghardware cause the data processing hardware to perform operationscomprising: filtering the infrared image data with a fluorescence imageenhancer having a first digital finite impulse response (FIR) filter anda second digital FIR filter different from the first FIR filter, and athird digital FIR filter configured to equalize for contrast loss from afluorescence diffusion and the second digital FIR filter is configuredwith a phase response to spatially align the white light image data withthe infrared image data; combining the white light image data and thefiltered infrared image data into combined image data; and transmittingthe combined image data to the display.
 22. The fluorescence imagingsystem of claim 21, wherein the first digital FIR filter is configuredto produce a magnitude response of zero at a horizontal Nyquistfrequency and a vertical Nyquist frequency and the second digital FIRfilter is configured with a phase response to spatially align the whitelight image data with the infrared image data.
 23. The fluorescenceimaging system of claim 21, wherein the third digital FIR filter isfurther configured to be gain limited to 20 dB.
 24. The fluorescenceimaging system of claim 21, wherein the first digital FIR filter isconfigured to equalize for contrast loss from fluorescence diffusion andthe second digital FIR filter is configured to adjust a sensitivity ofthe infrared image data and a contrast of the infrared image data basedon received infrared light.
 25. The fluorescence imaging system of claim24, wherein the second digital FIR filter is configured to adjust thesensitivity and the contrast of the infrared image data by adjusting anaccumulation of light in a neighborhood of the infrared image data. 26.The fluorescence imaging system of claim 24, wherein the second digitalFIR filter comprises a low pass filter with a cutoff frequency and amagnitude of a frequency response.
 27. The fluorescence imaging systemof claim 26, wherein the second digital FIR filter is further configuredto increase the sensitivity of the infrared image data and decrease thecontrast of the infrared image data when the received infrared light isbelow a threshold level.
 28. The fluorescence imaging system of claim27, wherein the threshold level is 10 microwatts.
 29. The fluorescenceimaging system of claim 27, wherein the second digital FIR filter isfurther configured to increase the sensitivity of the infrared imagedata by increasing the magnitude and decrease the contrast of theinfrared image data by decreasing the cutoff frequency when the receivedinfrared light is below a threshold level.
 30. The fluorescence imagingsystem of claim 27, wherein the second digital FIR filter is furtherconfigured to decrease the sensitivity of the infrared image data andincrease the contrast of the infrared image data when the receivedinfrared light is above a threshold level.
 31. The fluorescence imagingsystem of claim 27, wherein the third digital FIR filter is furtherconfigured to decrease the sensitivity of the infrared image data bydecreasing the magnitude and increase the contrast of the infrared imagedata by increasing the cutoff frequency when the received infrared lightis below a threshold level.
 32. The fluorescence imaging system of claim21, wherein the infrared image sensor and the white light image sensorhave the same resolution.
 33. The fluorescence imaging system of claim21, wherein the infrared image sensor comprises a near infrared imagesensor.
 34. The fluorescence imaging system of claim 21, wherein theinfrared image sensor and the white light image sensor comprise a Bayercolor filter array.
 35. The fluorescence imaging system of claim 21,further comprising a dichroic prism, wherein the dichroic prism splitsreceived light into white light and infrared light, and wherein theinfrared image sensor receives the infrared light from the dichroicprism, and wherein the white light image sensor receives the white lightfrom the dichroic prism.
 36. The fluorescence imaging system of claim22, wherein the first digital FIR filter is further configured toproduce a flat magnitude response prior to the horizontal Nyquistfrequency and the vertical Nyquist frequency.
 37. The fluorescenceimaging system of claim 22, wherein the second digital FIR filter isconfigured to spatially adjust a position of the infrared image data intwo dimensions.
 38. The fluorescence imaging system of claim 22, whereinthe second digital FIR filter is configured to adjust an image by atleast one of one half of a pixel horizontally or one half of a pixelvertically.